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Using the magnetization enumerator method, we evaluate the practical and theoretical limitations 
of symmetric channels with real outputs. Results are presented for several regular Gallager code 
constructions. 

Error correcting codes play a central role in modern communication, especially in noisy media such 
as in satellite and mobile communication. A broad range of error correcting codes are on offer; they 
vary significantly in their practical and theoretical performance depending on the specific code chosen 
within a given code ensemble. Evaluating the limitations of specific code constructions is important 
for determining the code efficiency and for optimizing channel performance. One of the leading code 
ensembles to date is the family of Low Density Parity Check Codes P, Q (LDPC), which attracted 
significant interest both within and outside of the information theory community. 

Methods of statistical physics have been recently employed to studythe typical performance of various 
coding schemes, most notably of Low Density Parity Check Codes 11,0, ISM Such studies have 

led to precise estimations of critical channel-noise levels (beyond which decoding is not possible) and 
also provided additional insight through the physical interpretation of various decoding schemes. The 
emerging picture for Gallager-type codes is that for sufficiently small noise levels decoding is possible and 
the error-free (ferromagnetic) state is the only solution. For higher noise levels one finds a transition to 
a regime where suboptimal solutions are created (spinodal or dynamical transition) and where existing 
practical decoding algorithms fail to find the most probable solution. For higher noise levels, a second 
transition occurs (thermodynamic transition) where the error-free solution ceases to be dominant. This 
marks the upper theoretical bound for error-free communication using the code specified. The number of 
equally plausible solutions to the decoding problem thereafter is exponential in the number of degrees of 
freedom. The thermodynamic transition approaches Shannon's limit with an increasing number of parity 
checks per bit. 

One of the most important aspects in the decoding problem is the enumeration of possible solutions, as 
it provides a direct indication to the practical and theoretical performance of various decoding methods. 
A method for carrying out the analysis have been reported recently in Q; the new approach generated 
interest in the application of the same method to other channel types characterized by real noise and real 
output values studied in this report. In particular we study the cases of the Gaussian and Laplace channels 
which are of high practical relevance, and are used as standard benchmark channels for evaluating the 
performance of codes. We consider the channels to be symmetric, i.e., the probabilities P{zoMt\zin) = 
-P(— ^outl — ^in), where Zout and Zjn are the input and output channel values respectively. We also consider 
information vectors representing binary messages t e {0, 1}^ to be encoded via Gallager's scheme. 

A Gallager code is defined by the binary parity check matrix A — [Ci IC2] of dimensionality {M—N)xM, 
which is a concatenation of two submatrices. A regular {K, C) Gallager code has a fixed number K of 
non-zero elements per row in A, and a fixed number C of non-zero elements per column, ft follows that 
C = K[M - N)/M. 

The message vector t e {0, 1}^ is encoded to the codeword x ~ G^t e {0, 1}*^ prior to transmission, 
using the generator matrix G — [l\{C2^Gi)'^'\ . This construction ensures that AG^ = (mod 2). 
Redundancy in the codeword, in the case of unbiased messages, is measured by the rate R = N/M = 
1 — C/K. After transmission of the codeword through the noisy channel, the following message is received 

y = X + (1) 

where G M^^ represents the real channel noise which, in the case of Gaussian and Laplace channels, 
has the distribution: 

Gaussian : P«) = J_ exp[-^{nl)^] (2) 



Laplace: P(„[) = _ exp[-^] , (3) 

per bit i. The channel noise level is measured by the parameters a and A respectively. 

Note that since the channel noise {n[} consists of real-valued variables (unlike the binary symmetric 
channel studied in 'B^ ) , the channel output y is also real- valued and the evaluation of the syndrome vector 
cannot be based on simple modulo 2 operations. Decoding is carried out by applying Bayes rule and 
the corresponding noise model to calculate P{x — ±l|j/) for each bit. To bring the problem back to a 
binary setting, we follow the procedure of and consider a fictitious channel where the sent message x 
is corrupted by binary noise G {0, 1}^^: 

r ^ x + n^ (mod 2) (4) 

Without loss of generality, one can take n-f — x such that r = and P{n^) — P{y\x) — P{y\n^). 

We denote the set of (fictitious) noise vectors n that satisfy the parity check equations An — Ar — 
by Ipc — {n\An — 0}, the parity check set. To infer the original message one needs to find the original 
fictitious noise from the parity check set on the basis of its statistics. The conditional probabilities of 
the fictitious noise variables (that satisfy the parity checks) follow from applying Bayes rule (per bit i) 

r>! I ^ P(.yi\ni)P{n,) 

It was shown (e.g., in 0,0]) that this problem can be cast into a statistical mechanical formulation 
replacing the field ({0, 1}, -|-mod 2) by the field ({1,-1}, x) and by suitably adapting the parity checks. 
Using the fact that rii are Ising variables with prior P{ni) = 1/2, can now be written as P{n\y) — 
exp(/3 i/(n)) where the energy H{n) is up to a constant given by 

i?(n) = ^ In P(n,Kj,) = i^n,/i„ = ^ ^ TlnP{y,\T) (6) 

i I Te{-i,i} 

and (3=1 (which corresponds to Nishimori's condition T^). To unify notation for the Gaussian and 
the Laplace channel, we denote the channel degradation parameter variable by d, where d = a^, X for 
the Gaussian and Laplace channel respectively. With these definitions, for any symmetric channel with 
real outputs, the local energies are staggered magnetizations along the fields hi, the distribution of which 
follows from ^ in combination with ||2Jl and ^ respectively: 

I 2~ 2 

Gaussian : p(/i,) = e^'^i'^^i^)^ (7) 



Laplace: p(h,) = - Sih, ~ X'^) + ^ d{h, + X'^) + e[X-' - \h,\] -e''^-^'" (8) 

where Q[x] is the Heavyside function returning 1 if x > and if x < 0. 

The entropy of the solutions to the decoding problem with a given magnetization m(n; h) = 

-M Y.i'^i^i = is 

M{m) = ^l\n 5[m-m{n-h)]\ . (9) 

\ neipa(n,nf;A) I 

Averages in Q are taken over the fields {hi\ or ((SJ), the parity check constructions A, whereas the 
original fictitious noise nf is gauged away using the transformations rn nin{ and yi — > yin{ . In order 
to perform the averages, we employ the replica identity: {\nM{m)) = lim„^o ^ ln(7W"(m)) . In the hmit 



n ^ and within the repHca symmetric assumption (shown to be exact in this case for obtaining the 
theoretical critical noise levels j^; for technical details also see e.g. [Hloj), we find that 

fee 1 

M{m) = Extr^_^_A <^ -mm - -^ln2 + j^hM - C/2[7r, tt] + /3[7r; m] ^ (10) 

with 

.if K 

hM = / l[{dxkTT{xk)} ln[l + Y[ Xk] (11) 

k=l k=l 



dxdx 'k{x)tt{x) ln(l + xa;) (12) 



•' c=l \ r=± c=l / ^ 

The functional extremization problem in (|10|l results in the following saddle point equations 

A'-l K-l 

T^{x) = / H {dxk'K{xk)} 5[x - Jl Xk] (14) 

^ I -I ; 1 



fc = l fc=l 

. C-1 



7r(a:) = y Jl {dxc7r(£c)} 



c-i 



— tanh[7fi/i + atanh(ic)] 



c=i J / ft, 



(15) 



c=l 

Equations H14|) and H15|l are the infinite system equiyalent of the so-called density-evolution equations , 
and iteratively converge to the stationary distributions tt* {x) , n* (x) . Equations H1()I16|I are then evaluated 
for these stationary distributions. One should note that the solution tt*{x) = S[x—1] and 7r*(a;) = 5[a; — 1] 
always exists, and has a magnetization mo = {h)h and zero entropy. This (ferromagnetic) solution 
corresponds to perfect retrieval (n-^ after the gauge), and should be compared to alternative solutions, if 
they exist, which correspond to the other (sub-optimal) noise candidates in Xpc{n, A). 

In the limit K,C^oo (while keeping the rate R finite) it is possible to derive these alternative solutions 
for all values of m, analytically. This case is, however, of little practical interest, and will not be discussed 
in this paper. 

For finite K, C, alternative analytic solutions can no longer be obtained for both Gaussian and Laplace 
channels, and one has to solve the saddle point equations numerically to obtain M(m). 

The critical noise values of the thermodynamic and spinodal transitions now follow directly from the 
graphs of Ai (m) , for different values of d. 

As explained in full detail in 0, the theoretical critical degradation value d — dc is reached, for 
Maximum A Posteriori (MAP) and typical set decoding methods, when the magnetization at which the 
entropy of the sub-optimal solutions vanishes, coincides with that of the ferromagnetic solution. For 
finite temperature decoding at the Nishimori temperature the critical noise value coincides with the 
thermodynamic transition at which the ferromagnetic solution ceases to be dominant, and A4{m) has a 
slope dM{m)/dm\^^^f^-^^ = —1/d at m = mg = {h)h (see Fig.^l. 

Of more practical interest is the limiting practical noise level d — dd, above which practical algorithms 
such as density evolution Q break down. This transition is signaled by the emergence of suboptimal 
solutions for eqns. H14I15|I . These correspond to local minima of the free energy in which the algorithm 
gets trapped; this is known as a spinodal point or dynamical transition. The noise level d — dd can 
be obtained from Ai{m)] it is the smallest positive value d for which there exists a value m* such 



4 



that dA4{m) / dm\^, = —1/d. This typically happens at a value of m* for which A4{m*) < 0, such 
that the practical transition point is upper-bounded by the thermodynamic transition point, which is 
in turn upper-bounded by Shannon's information-theoretic limit, and the following inequalities hold: 
dd< dc < ds- 

In Fig.^we present the magnetization enumerator for the Gaussian and Laplace channels for a {K, C) = 
(6, 3) code at the thermodynamic transition. The maximum number of solutions to both channels depends 
only on the code rate R as A^(0) — i?ln2. 

It should be noted that values of M. (m) < are unphysical, and are an artifact of the replica symmetric 
assumption. Nevertheless, this region turns out to be relevant for the determination of the dynamical 
(spinodal) transition. This can be understood by the fact that the replica symmetric fixed point equations 
are the exact infinite size equivalent of the practical density evolution equations 2]. Therefore, although 
quantities related to the M.{m) < region, are unphysical and should be corrected by a refined replica 
symmetry ansatz, the region AA(rn) < and the fixed point equations associated with it, still allow us 
to determine the dynamical transition point. 

The calculated critical noise levels are presented in the Table HI for several code constructions and 
for the Gaussian and Laplace channels. These values are in excellent agreement with those obtained 
independently in Q using a different method, as well as with the corresponding practical upper-bounds 
of |2| obtained using density evolution. Note that we have also presented the critical degradation values 
for different code constructions of identical rates, to illustrate the opposite tendencies for the theoretical 
and practical critical values with increasing K and C. Increasing K and C (keeping K/C fixed) pushes 
the thermodynamic critical value closer to Shannon's information-theoretic limit, but adversely affects 
the practically admissible degradation value. This is in agreement with the common belief that code 
constructions with higher connectivity are less practical. 
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FIG. 1: The magnetization enumerator for the Gaussian (solid curve) and the Laplace (dashed curve) channels 
for a {K, C) = (6, 3) code at the thermodynamic transition noise levels (cr^ = 0.899 , Ac = 0.712). For both 
channels the maximum number of solutions is M{0) = i?ln2. The energy of solutions is given by E{m) = — ^rn, 
while their free energy at the Nishimori temperature T^l is up to a constant given by the orthogonal distance 
to the straight hues. At the thermodynamic transition point, these are tangents to M(m) at m* = mo(=l and 
0.665 for the two channels respectively). 

In this report we have shown how the magnetization enumerator formalism can be easily extended 
to real-valued channels, in order to obtain both theoretical and practical critical values for the degrada- 
tion parameter. Following the method presented in we have mapped the real valued channel onto an 
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(K,C) 


R 


Ad 


Ac 


As 


(6,3) 


0.5 


0.775 


0.899 


0.958 




(6,3) 


0.5 


0.651 


0.712 


0.752 


(5,3) 


0.4 


1.017 


1.253 


1.321 




(5,3) 


0.4 


0.773 


0.875 


0.914 


(6,4) 


0.333 


1.020 


1.666 


1.681 




(6,4) 


0.333 


0.782 


1.045 


1.055 


(9,6) 


0.333 


0.379 


1.679 


1.681 




(9,6) 


0.333 


0.661 


1.048 


1.055 


(4,3) 


0.25 


1.598 


2.325 


2.401 




(4,3) 


0.25 


1.018 


1.260 


1.298 


(8,6) 


0.25 


0.880 


2.396 


2.401 




(8,6) 


0.25 


0.619 


1.271 


1.298 



TABLE L Values of the critical noise levels (spinodal and thermodynamic transitions) of the Gaussian and Laplace 
channels for various regular {K, C) Gallager codes. For comparison, in the last column we present Shannon's 
information-theoretic bound. 



equivalent fictitious binary channel, and employed methods of statistical physics to calculate the magne- 
tization enumerator for the Gallager code ensemble. The magnetization enumerator is instructive in the 
way it nicely links the various decoding schemes 5] and facilitates the derivation of both practical and 
theoretical critical noise levels. 

Using Nishimori's gauge theory, the theoretical critical noise levels can be shown to be exact, while our 
practical critical degradation parameters, are in excellent agreement with those obtained known in the 
literature when available. 

Studying the magnetization enumerator further, beyond the practical limiting noise level may provide 
additional insight into the decoding complexity the performance of Gallager type codes. 
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